Multi-stage training and knowledge transfer, from a large-scale pretraining task to various finetuning tasks, have revolutionized natural language processing and computer vision resulting in state-of-the-art performance improvements. In this paper, we develop a multi-stage influence function score to track predictions from a finetuned model all the way back to the pretraining data. With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task. The proposed multi-stage influence function generalizes the original influence function for a single model in (Koh & Liang, 2017), thereby enabling influence computation through both pretrained and finetuned models. We study two different scenarios with the pretrained embedding fixed or updated in the finetuning tasks. We test our proposed method in various experiments to show its effectiveness and potential applications.

My main concern is that the authors did not compare the proposed method with any latent baselines. For example, we can also use to the uncertainty (the loss value or the entropy of the predicted distribution) of the model as an indicator to identify those problematic examples in the pre-training data. As the proposed method in this paper does not show very impressive results in the experiments (the Pearson's correlation is only 0.4 0.6 in Figure 1), it may not outperform this simple baseline. "At the pretraining stage, we train the models with examples from two classes ("bird" vs. "frog") for CIFAR-10 and four classes (0, 1, 2, and 3) for MNIST". The transfer tasks in these settings may be too easy.

Multi-stage training and knowledge transfer, from a large-scale pretraining task to various finetuning tasks, have revolutionized natural language processing and computer vision resulting in state-of-the-art performance improvements. In this paper, we develop a multi-stage influence function score to track predictions from a finetuned model all the way back to the pretraining data. With this score, we can identify the pretraining examples in the pretraining task that contribute most to a prediction in the finetuning task. The proposed multi-stage influence function generalizes the original influence function for a single model in (Koh & Liang, 2017), thereby enabling influence computation through both pretrained and finetuned models. We study two different scenarios with the pretrained embedding fixed or updated in the finetuning tasks.